Image processing apparatus and method using tracking of gaze of user

ABSTRACT

An apparatus and method for outputting view images by tracking a gaze of a user are provided. The image processing apparatus estimates movement of a gaze of a user using a camera, and determines an output order of view images according to the gaze movement.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Korean Patent Application No.10-2010-0087549, filed on Sep. 7, 2010, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference.

BACKGROUND

1. Field

Example embodiments of the present disclosure relate to an imageprocessing apparatus and method capable of outputting view images bytracking a gaze of a user.

2. Description of the Related Art

To effectively implement a stereoscopic effect of a 3-dimensional (3D)image, images from different viewpoints should be supplied to each ofthe left and right eyes of a person. To achieve the 3D effect without afilter such as dedicated glasses, a 3D image needs to be expressed in astate of being separated spatially based on viewpoints, which is calledan autostereoscopic display. In an autostereoscopic display, an image isdisplayed in a space in a state of being separated by an optical devicesuch as an optical lens, a barrier, and the like. When using the opticallens, the autostereoscopic display uses a lenticular lens so that pixelimages are expressed only in a specific direction. When using theoptical barrier, only a specific pixel can be seen in a specificdirection since a slit is placed in front of the autostereoscopicdisplay. As described above, the autostereoscopic display using theoptical lens or optical barrier expresses images from two viewpoints,that is, a right viewpoint and a left viewpoint, and a sweet spot havinga significantly narrow stereoscopic viewing angle is generated. Thesweet spot is expressed by a viewing distance and a viewing angle. Theviewing distance is determined by a pitch of a lens or a slit. Theviewing angle is determined by the number of expressed viewpoints. Whenthe viewing angle is widened by increasing the number of viewpoints, anautostereoscopic multi-view display is achieved.

A wide viewing area may be achieved by the autostereoscopic multi-viewdisplay. However, since 3D resolution is proportional to the number ofviewpoints, the autostereoscopic multi-view display reduces resolutionof the displayed image.

Accordingly, there is a demand for a method capable of providing a wideviewing area by increasing the number of viewpoints while minimizingreduction in the resolution caused by the increased number ofviewpoints.

SUMMARY

According to one or more example embodiments of the present disclosure,an image processing apparatus may be provided, which includes a movementestimation unit to estimate a movement of a gaze of a user, and anoutput order determining unit to determine an output order of pluralityof view images corresponding to the movement of the gaze of the user.

The movement estimation unit may estimate a movement direction of thegaze of the user. Therefore, the output order determining unit mayadjust the output order of the plurality of view images corresponding toa right eye of the user when the gaze of the user is moved to the left.

The output order determining unit may adjust the output order of theplurality of view images corresponding to a left eye of the user whenthe gaze of the user is moved to the right.

The movement estimation unit may estimate a movement angle of the gazeof the user. Therefore, the output order determining unit may vary thenumber of the plurality of view images of which the output order is tobe adjusted, based on the movement angle.

As the movement angle increases, the output order determining unit mayincrease the number of the plurality of view images of which the outputorder is to be adjusted.

The image processing apparatus may further include a display to displayan output image comprising a plurality of the view images, according tothe determined output order.

The output image may include pixel units each containing at least oneview image.

The output order determining unit may determine the output order of theplurality of view images such that a predetermined number of left andright view images are consecutively arranged with respect to a referenceview image, the reference view image corresponding to a position towhich the gaze of the user is moved.

According to example embodiments, there may be also provided an imageprocessing method including estimating a movement of a gaze of a user;and determining an output order of plurality of view imagescorresponding to the movement of the gaze of the user.

The estimating of the gaze movement may estimate a movement direction ofthe gaze of the user, and the determining of the output order may adjustan output order of the plurality of view images corresponding to a righteye of the user when the gaze of the user is moved to the right.

The determining of the output order may adjust an output order of theplurality of view images corresponding to a left eye of the user whenthe gaze of the user is moved to the left.

The estimating of the gaze movement may estimate a movement angle of thegaze of the user. Therefore, the determining of the output order mayvary the number of the plurality of view images of which the outputorder is to be adjusted, according to the movement angle.

The determining of the output order may increase the number of pluralityof view images of which the output order is to be adjusted as themovement angle increases.

The image processing method may further include displaying an outputimage comprising a plurality of the view images according to thedetermined output order.

The determining of the output order may determine the output order ofthe plurality of view images such that a predetermined number of leftand right view images are consecutively arranged with respect to areference view image, the reference view image corresponding to aposition to which the gaze of the user is moved.

Additional aspects, features, and/or advantages of example embodimentswill be set forth in part in the description which follows and, in part,will be apparent from the description, or may be learned by practice ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages will become apparent and morereadily appreciated from the following description of the exampleembodiments, taken in conjunction with the accompanying drawings ofwhich:

FIG. 1 illustrates a diagram for explaining a case where an inversionphenomenon occurs due to a movement of a gaze of a user;

FIG. 2 illustrates a block diagram showing a structure of an imageprocessing apparatus according to example embodiments;

FIG. 3 illustrates a diagram for describing a process of calculating amovement distance;

FIG. 4 is a diagram illustrating view images corresponding to a gaze ofa user when an input image is a 36-view image and the user is located ina middle of a display;

FIGS. 5 through 7 illustrate a process of determining an output order ofview images when an input image is a 36-view image and a gaze of a useris moved to the right;

FIGS. 8 through 10 illustrate a process of determining an output orderof view images when an input image is a 36-view image and a gaze of auser is moved to the left;

FIG. 11 is a diagram illustrating view images corresponding to a gaze ofa user when an input image is a 12-view image and the user is located ina middle of a display;

FIGS. 12 through 14 illustrate a process of determining an output orderof view images when an input image is a 12-view image and a gaze of auser is moved to the right;

FIGS. 15 through 17 illustrate a process of determining an output orderof view images when an input image is a 12-view image and a gaze of auser is moved to the left;

FIG. 18 illustrates a process of determining an output order of viewimages using only 2 view images when an input image is a 36-view imageand a gaze of a user is moved to the right;

FIG. 19 illustrates a process of determining an output order of viewimages using only 2 view images when an input image is a 36-view imageand a gaze of a user is moved to the left;

FIG. 20 illustrates a process of determining an output order of viewimages using only 2 view images when an input image is a 12-view imageand a gaze of a user is moved to the right;

FIG. 21 illustrates a process of determining an output order of viewimages using only 2 view images when an input image is a 12-view imageand a gaze of a user is moved to the left; and

FIG. 22 is a flowchart for describing a process of determining an outputorder of view images according to movement of a gaze of a user.

DETAILED DESCRIPTION

Reference will now be made in detail to example embodiments, examples ofwhich are illustrated in the accompanying drawings, wherein likereference numerals refer to the like elements throughout. Exampleembodiments are described below to explain the present disclosure byreferring to the figures.

FIG. 1 illustrates a diagram for describing a case where an inversionphenomenon occurs according to movement of a gaze of a user 30.

When the user moves to the right, a left eye L of the user is disposedin an orthoscopic viewing zone of a display 10 and a right eye R isdisposed in a sub-viewing zone of the display 10. Accordingly, a rightview image is seen by the left eye L while a left view image is seen bythe right eye R, which may make the user 30 feel dizzy.

For example, according to FIG. 1, a sensing device 20 may sense the lefteye L of the user gazing at a 2-view image and the right eye R gazing ata 10-view image. When the gaze of the user is moved to the right, theright eye R is disposed in the sub-viewing zone, thereby gazing at a1-view image. Therefore, the right eye R that has been gazing at the10-view image which is a right view image comes to gaze at the 1-viewimage which is a left view image. As a result, an inversion phenomenonoccurs to the right eye R. Here, the inversion may be prevented bychanging an output order of the view images in such a manner that a12-view image is displayed in a position where the 1-view image of theuser 30 is to be disposed. That is, the inversion may be prevented byperforming rendering such that the 12-view image is disposed in theposition where the 1-view image is to be disposed.

Hereinafter, a process of determining the output order of view imagesaccording to movement of a gaze of a user will be further described withreference to FIG. 2.

FIG. 2 is a block diagram illustrating a structure of an imageprocessing apparatus 200 according to example embodiments.

Referring to FIG. 2, the image processing apparatus 200 may include, forexample, a movement estimation unit 210, an output order determiningunit 220, and a display 230.

The movement estimation unit 210 may estimate whether the gaze of theuser is moved, based on user eye position information sensed by asensing device 100. Here, the movement estimation unit 210 may include amovement distance calculating unit 211 and a movement angle calculatingunit 212.

For example, the sensing device 100 may photograph the user in real timeand transmit a photographed image to the movement distance calculatingunit 211 in real time. The sensing device 100 may include any one of aweb camera, a monocular camera, a stereo camera, a multi camera, and acamera measuring depth-information. Therefore, the movement distancecalculation unit 211 is capable of calculating a position of left orright eyes or both eyes of the user from the photographed image. Theuser eye position may be expressed by a coordinate value correspondingto view images being displayed on the display 230. In other words, theuser eye position may be expressed by a coordinate value of the viewimages corresponding to positions gazed upon by the eyes of the user.For example, the user eye position calculated from the photographedimage may be (x1, y1).

When a next photographed image is received, the movement distancecalculating unit 211 may calculate a next position of the left and righteyes of the user from the next photographed image. For example, the nextuser eye position may be (x2, y2).

In addition, the movement distance calculating unit 211 may determine adirection in which the gaze of the user is moved, that is, a movementdirection of the gaze, by calculating a left difference value (x2-x1)between a current left eye position and a next left eye position and aright difference value (y2-y1) between a current right eye position anda next right eye position. Here, when any one of the left differencevalue and the right difference value is equal to or greater than apreset error value, the movement distance calculating unit 211 mayestimate that the gaze is moved. In addition, when the left and theright difference values are positive, the movement distance calculatingunit 211 may estimate that the gaze is moved to the right. When the leftand right difference values are negative, the movement distancecalculating unit 211 may estimate that the gaze is moved to the left.

Referring to FIG. 3, additionally, the movement distance calculatingunit 211 may calculate a coordinate value comprised of the left and theright difference values as a movement distance x. For example, themovement distance x may be expressed as a coordinate value (x2-x1,y2-y1).

The movement angle calculating unit 212 may calculate a movement angleof an eye of the user, corresponding to the gaze movement of the user,using the movement distance x and a focal distance fl. Here, a focaldistance of the sensing device 100 may be preset as the focal distancefl. The movement angle calculating unit 212 may calculate the movementangle of the eye using Equation 1 below.

$\begin{matrix}{\theta = {\tan^{- 1}\left( \frac{x}{fl} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

wherein, θ refers to the movement angle, and fl refers to the focaldistance.

Referring to Equation 1, the movement angle calculating unit 212 maycalculate the movement angle by dividing the movement distance by thefocal distance and calculating an arc tangent value of the quotient.Therefore, the output order determining unit 220 may determine theoutput order of the view images based on the calculated movement angle.

More specifically, the output order determining unit 220 may determinethe output order of the view images corresponding to the movement of thegaze. When the gaze is estimated to have moved, the output orderdetermining unit 220 may vary the number of the view images of which theoutput order is to be adjusted, in accordance with the calculatedmovement angle.

For example, referring to FIG. 4, when an input image is a 36-view imageand when a display being used is capable of displaying 12 view images,the left eye of the user gazes at a 22-view image 421 and the right eyegazes at a 15-view image. Accordingly, a 3D image is seen by the user.

As the gaze of the user moves to the right as shown in FIG. 5, the lefteye gazes at a 21-view image 522 and the right eye gazes at a 14-viewimage 524. Therefore, the movement angle calculating unit 212 estimatesthat the right eye is moved to the right by about 2θ and, accordingly,the output order determining unit 220 may adjust the output order of theview images corresponding to the right eye, based on the calculatedmovement angle.

Here, when a lens of the display 400 is tilted by a predetermined angle,view images neighboring on the left and the right may be simultaneouslydisplayed by a single pixel. For example, in a display that displays 12view images, one pixel may display 5 view images at once. Therefore, asshown in FIG. 5, pluralities of views included in left and right slidewindows 530 and 540, respectively, need to be simultaneously displayedto achieve a natural 3D image. To achieve a natural 3D image, the outputorder determining unit 220 may adjust an output order of the view imagesas shown in FIG. 5. Therefore, the user gazes at the view images beingdisplayed according to the output order 520 as adjusted by a display500. For example, even though the gaze of the user is moved to the rightby about 1θ, the view images are seen by the right eye in order of 16,15, 14, 13, and 12-view images. Thus, when the output order of the viewimages is adjusted according to the movement of the gaze, a 12-viewimage 525, instead of a 24-view image, is continued next to the 13-viewimage, thereby preventing the inversion.

In a similar manner, as shown in FIG. 6, when the gaze of the user ismoved to the right by about 2θ, the left eye gazing at a 22-view image621 may be moved to a position for gazing at a 20-view image 622, andthe right eye gazing at a 15-view image 623 may be moved to a positionfor gazing at a 13-view image 624. Here, the output order determiningunit 220 may determine the output order of the view images such thatview images continued to the left and the right of a view imagecorresponding to the position to which the gaze is moved are naturallycontinued. That is, as the right eye of the user is moved to the rightby about 2θ, the output order determining unit 220 may determine theoutput order such that the view images are seen by the right eye inorder of 15, 14, 13, 12, and 11-view images instead of in order of 15,14, 13, 24, and 23-view images. Therefore, the view images 12 and 11 625are continued instead of the view images 24 and 23 next to the 13-viewimage 624. As a result, the user is able to see a natural 3D image.

In the same manner, referring to FIGS. 4 and 7, when the gaze of theuser is moved to the right by about 3θ, the left eye gazing at a 22-viewimage 421, as illustrated in FIG. 4, may be moved to a position forgazing at a 19-view image 722, as illustrated in FIG. 7, and the righteye gazing at a 15-view image 723 may be moved to a position for gazingat a 12-view image 724 as illustrated in FIG. 7. Here, the output orderdetermining unit 220 may determine the output order of the view imagesconstituting an output image such that pluralities of the view imagesincluded in the left and the right slide windows 730 and 740 arenaturally continued along the movement of the gaze. For example, theoutput order determining unit 220 may determine to output the viewimages in order of 13, 14, 16, 17, 18, 19, 20, 21, 10, 11, and 12-viewimages 710. Accordingly, 12, 11, and 10-view images 725 instead of theview images 24, 23 and 22 may be consecutively seen by the right eyeafter the 13-view image.

As described above, as the movement angle increases from θ to 2θ and 3θ,the number of the view images of which the output order is to beadjusted may be increased from 1 to 2 and from 2 to 3, respectively.

In the above, the process of determining the output order of the viewimages corresponding to the movement of the gaze when the gaze is movedto the right has been described with reference to FIGS. 4 through 7. InFIGS. 5 through 7, the output order determining unit 220 may achieve anatural 3D image by determining the output order such that the viewimages that can be simultaneously displayed in one pixel areconsecutively arranged. In other words, the output order maybedetermined such that a plurality of the view images of the left slidewindow are consecutive left view images and a plurality of the viewimages of the right slide window are consecutive right view images.

Here, among the plurality of view images, the view image correspondingto the position of the gaze may be disposed in the middle of the slidewindow. The same number of view images is arranged on the left and theright of the view image disposed in the middle of the slide window.Referring to FIG. 7, for example, as the gaze is moved to the right byabout 3θ, the 12-view image 724 corresponding to the gaze is disposed inthe middle of the slide window while a preset reference number of viewimages are continued to the left and to the right of the 12-view image724, respectively. When the reference number is set to 2, the 14 and13-view images may be continued to the left of the 12-view image 724 andthe 11 and 10-view images may be continued to the right of the 12-viewimage 724. The reference number may be preset as a value obtained bydividing the number of view images simultaneously displayed in one pixelby 2.

As described above, the output order determining unit 220 may determinethe output order of the plurality of view images constituting the outputimage based on the movement angle calculated as the gaze of the user ismoved. Accordingly, the display 230 may display the plurality of viewimages according to the determined order. Here, the output orderdetermining unit 220 may determine the output order using a databasewhere the output orders according to the movement angles are stored.

Table 1 shows the output orders corresponding to the movement anglescalculated as the gaze of the user is moved to the right. The inputimage in Table 1 is a 36-view image.

TABLE 1 Movement angle Output orders of view images (θ) 1 2 3 4 5 6 7 89 10 11 12 0 13 14 15 16 17 18 19 20 21 22 23 24 1 13 14 15 16 17 18 1920 21 22 23 22 2 13 14 15 16 17 18 19 20 21 22 11 12 3 13 14 15 16 17 1819 20 21 10 11 12 4 13 14 15 16 17 18 19 20 9 10 11 12 5 13 14 15 16 1718 19 8 9 10 11 12 6 13 14 15 16 17 18 7 8 9 10 11 12 7 13 14 15 16 17 67 8 9 10 11 12 8 13 14 15 16 5 6 7 8 9 10 11 12 9 13 14 15 4 5 6 7 8 910 11 12 10 13 14 3 4 5 6 7 8 9 10 11 12 11 13 2 3 4 5 6 7 8 9 10 11 1212 1 2 3 4 5 6 7 8 9 10 11 12

Referring to Table 1, the output order determining unit 220 maydetermine the output order according to the movement angles calculatedby the movement angle calculating unit 212. In addition, the display 230may display a natural 3D image despite the rightward movement of thegaze, by outputting the plurality of view images according to thedetermined output order. Here, when the movement angle is 0 in Table 1,the user is located in the middle of the display 230. When the movementangle is 12θ, both the left and the right eyes are moved to thesub-viewing zone. Therefore, the display 230 may display the outputimage comprised of 1 to 12-view images in that order.

The image processing apparatus 200 may enlarge a viewing angle twice asthe gaze of the user is moved to the right. In this case, the outputorder determining unit 220 may determine the output order of theplurality of view images according to Table 2 below. The display 230 maydisplay the plurality of view images according to the determined outputorder. Here, calculation of the movement angle according to the gazemovement is the same as described above regarding the movement anglecalculating unit 212 and therefore will not be described again.

Table 2 shows the output orders of the view images, the output orderscorresponding to the movement angles as the viewing angle increases. InTable 2, the input image is a 36-view image image.

TABLE 2 Movement angle Output orders of view images (θ) 1 2 3 4 5 6 7 89 10 11 12 0 13 14 15 16 17 18 19 20 21 22 23 24 1 13 14 15 16 17 18 1920 21 22 23 22 2 13 14 15 16 17 18 19 20 21 22 11 12 3 13 14 15 16 17 1819 20 21 10 11 12 4 13 14 15 16 17 18 19 20 9 10 11 12 5 13 14 15 16 1718 19 8 9 10 11 12 6 13 14 15 16 17 18 7 8 9 10 11 12 7 13 14 15 16 17 67 8 9 10 11 12 8 13 14 15 16 5 6 7 8 9 10 11 12 9 13 14 15 4 5 6 7 8 910 11 12 10 13 14 3 4 5 6 7 8 9 10 11 12 11 13 2 3 4 5 6 7 8 9 10 11 1212 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 1 14 1 2 3 4 56 7 8 9 10 1 1 15 1 2 3 4 5 6 7 8 9 1 1 1 16 1 2 3 4 5 6 7 8 1 1 1 1 171 2 3 4 5 6 7 1 1 1 1 1 18 1 2 3 4 5 6 1 1 1 1 1 1 19 1 2 3 4 5 1 1 1 11 1 1 20 1 2 3 4 1 1 1 1 1 1 1 1 21 1 2 3 1 1 1 1 1 1 1 1 1 22 1 2 1 1 11 1 1 1 1 1 1 23 1 1 1 1 1 1 1 1 1 1 1 1

According to Table 2, when the gaze of the user is moved to the right, awide viewing zone may be supplied without generating the inversion.

Table 3 shows the output orders of the view images, corresponding to themovement angles calculated according to a leftward movement of the gaze.In Table 3, the input image is a 36-view image.

TABLE 3 Movement angle Output orders of view images (θ) 1 2 3 4 5 6 7 89 10 11 12 0 13 14 15 16 17 18 19 20 21 22 23 24 1 25 14 15 16 17 18 1920 21 22 23 24 2 25 26 15 16 17 18 19 20 21 22 23 24 3 25 26 27 16 17 1819 20 21 22 23 24 4 25 26 27 28 17 18 19 20 21 22 23 24 5 25 26 27 28 2918 19 20 21 22 23 24 6 25 26 27 28 29 30 19 20 21 22 23 24 7 25 26 27 2829 30 31 20 21 22 23 24 8 25 26 27 28 29 30 31 32 21 22 23 24 9 25 26 2728 29 30 31 32 33 22 23 24 10 25 26 27 28 29 30 31 32 33 34 23 24 11 2526 27 28 29 30 31 32 33 34 35 24 12 25 26 27 28 29 30 31 32 33 34 35 36

Referring to Table 3, when the movement angle is 12θ, both the left andthe right eyes are disposed in the sub-viewing zone. Accordingly, theoutput order determining unit 220 may determine such that the viewimages constituting the output image are output in order of the 25 to36-view images instead of in order of the 13 to 24-view images. That is,the display 230 may display the output image rendered with the 25 to36-view images. Therefore, the output order determining unit 220 maydetermine the output order such that the user first sees the 13 to24-view images in the middle of the display 230 and further sees up tothe 36-view image while movement the gaze to the left. In other words,the viewing angle may be expanded to the left while preventing theinversion.

Referring to Table 3, in addition, the movement angle calculating unit212 may calculate the movement angle according to leftward movement ofthe gaze. In addition, the output order determining unit 220 maydetermine the output order of the plurality of view images by referringto Table 2. Therefore, the display 230 outputs the plurality of viewimages according to the determined output order, thereby displaying anatural 3D image even though the gaze of the user is moved leftward.

For example, in a state where the left eye gazes at a 22-view image 421and the right eye gazes at a 15-view image 422 among the plurality ofview images 410 as shown in FIG. 4, when the gaze is moved to the leftby about θ, the output order determining unit 220 may adjust the outputorder of the view images 410 of FIG. 4 to an order of the 25, 14, 15,16, 17, 18, 19, 20, 21, 22, 23, and 24-view images 810, as shown in FIG.8. Accordingly, a 25-view image 825, as shown in FIG. 8, instead of the13-view image is continued next to the 24-view image. Therefore, eventhough the gaze of the user is moved to the left by about 1θ, the lefteye and the right eye of the user may see a natural 3D image. Inparticular, the output order determining unit 220 may determine theoutput order such that the same number of view images is continued toboth the left and the right of a view image 824 corresponding to aposition of the left eye. For example, the 25 and 24-view images may becontinued to the left of the 23-view image while the 22 and 21-viewimages are continued to the right. Similarly, with respect to the16-view image, the 18 and 17-view images may be continued to the leftwhile the 15 and 14-view images are continued to the right. Thus, sincethe output order determining unit 220 determines the output order suchthat a left slide window 840 includes the left view images and a rightslide window 830 includes the right view images, the natural 3D imagemay be implemented without discontinuity even if the user's eyes areminutely moved to the left and the right.

Similarly, as shown in FIG. 9, when the gaze of the user is moved to theleft by about 2θ, the output order determining unit 220 may determine tooutput the view images in order of 25, 26, 15, 16, 17, 18, 19, 20, 21,22, 23, and 24-view images 910, according to Table 3. As a result, the26 and 25-view images 925 instead of the 14 and 13-view images may becontinued next to the 24-view image.

Also, as shown in FIG. 10, when the gaze of the user is moved to theleft by about 3θ, the output order determining unit 220 may determine tooutput the view images in order of 25, 26, 27, 16, 17, 18, 19, 20, 21,22, 23, 24-view images 1010, according to Table 3. As a result, the 15,14, and 13-view images instead of 27, 26, and 25-view images 1025 may becontinued next to the 24-view image.

A process of determining the output order of the view imagescorresponding to the gaze movement of the user when the input image is a36-view image and the display is capable of displaying 12 view imageswas previously described with reference to FIGS. 4 through 10.Hereinafter, a process of determining the output order of the viewimages corresponding to the gaze movement of the user when the inputimage is a 12-view image and a display is capable of displaying 12 viewimages is described.

FIG. 11 is a diagram illustrating view images corresponding to a gaze ofa user when an input image is a 12-view image and the user is located ina middle of a display 1100. Here, the display 1100 that displays 12 viewimages and the input image comprised of 12 views are used.

Referring to FIG. 11, when the display 1100 includes a lens, the viewimages may be seen by the user in the opposite order of the input imageowing to the characteristics of the lens. For example, when the viewimages are displayed in order from 1 to 12, the eyes of the user may seethe view images in order from 12 to 1. When the user is located in themiddle of the display 1100 in this state, the left eye of the user maybe gazing at a 10-view image 1121 while the right eye may be gazing at a3-view image 1122.

Here, as shown in FIG. 12, when the gaze of the user is moved to theright by about 1θ, the output order determining unit 220 may determineto output the view images in order of 01, 02, 03, 04, 05, 06, 07, 08,09, 10, 11, and 01. Accordingly, the left eye may gaze at a 9-view image1222 while the right eye gazes at a 02-view image 1224. In addition, a01-view image 1225 instead of a 12-view image may be continued next to a01-view image. Here, in the same manner as when the input image is the36-view image, the plurality of view images of a right slide window 1230are comprised of the right view images whereas the plurality of viewimages of a left slide window 1240 are comprised of the left viewimages. Sizes of the slide windows may be preset corresponding to thenumber of view images that can be simultaneously displayed in one pixel.Accordingly, the output image may include pixel units each including atleast one view image.

Similarly, when the gaze is moved by about 2θ as shown in FIG. 13, theoutput order determining unit 220 may determine to output the viewimages in order of 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 01, and01-view images. Accordingly, the display 230 may display the view images1310 according to the determined order. Therefore, even though the gazeof the user is moved to the right by about 3θ, the right eye may see 01and 01-view images 1330 instead of 12 and 11-view images continuingafter a 01-view image 1324. That is, the inversion may be prevented.

In the same manner, when the gaze is moved by about 3θ as shown in FIG.14, the output order determining unit 220 may determine to output theview images in order of 01, 02, 03, 04, 05, 06, 07, 08, 09, 01, 01, and01-view images. Accordingly, the display 230 may display the view images1410 according to the determined order. Therefore, even though the gazeof the user is moved to the right by about 3θ, the right eye may see 01,01, and 01-view images 1430 instead of 12, 11, and 10-view imagescontinuing after a 01 view image 1424. That is, the inversion may beprevented.

FIGS. 15 through 17 illustrate a process of determining an output orderof view images when an input image is a 12-view image and a gaze of auser is moved to the left. In FIGS. 15 through 17, the input image is a12-view image.

Referring to FIG. 15, when the gaze of the user is moved to the left byabout 1θ, the output order determining unit 220 may determine to outputthe view images in order of 12, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11,and 12-view images. Accordingly, the display 230 may display the viewimages 1510 according to the determined order. Therefore, even thoughthe gaze is moved to the left by about 1θ, the left eye may see a12-view image 1525 instead of a 01-view image continuing after another12-view image. That is, the inversion may be prevented.

Similarly, when the gaze of the user is moved to the left by about 2θ asshown in FIG. 16, the output order determining unit 220 may determine tooutput the view images in order of 12, 12, 03, 04, 05, 06, 07, 08, 09,10, 11, and 12-view images. Accordingly, the display 230 may display theview images 1610 according to the determined order. Therefore, eventhough the gaze is moved to the left by about 2θ, the left eye may seetwo instances of a 12-view image 1625 instead of 01 and 02-view imagescontinuing after a 12-view image 1622. That is, the inversion may beprevented.

Similarly, when the gaze of the user is moved to the left by about 3θ asshown in FIG. 17, the output order determining unit 220 may determine tooutput the view images in order of 12, 12, 12, 04, 05, 06, 07, 08, 09,10, 11, and 12-view images. Accordingly, the display 230 may display theview images 1710 according to the determined order. Therefore, eventhough the gaze is moved to the left by about 3θ, the left eye may see12, 12, and 12-view images 1625 instead of 03, 02 and 01-view imagescontinuing after a 12-view image. As a result, the inversion may beprevented. That is, a left slide window 1740 may include the 12, 12, and12-view images 1725 which are the left view images instead of the 03,02, and 01-view images which are the right view images.

The process of displaying a natural 3D image by slide windows thatinclude different view images has been described above. Hereinafter, aprocess of selecting two view images from the plurality of view imagesconstituting the input image and determining the output order of theview images using the selected view images is described. In this case,deterioration of a 3D image by a crosstalk may be prevented byperforming rendering using the selected two view images in a positionwhere 12-view images are to be displayed.

For example, the output order determining unit 220 may output order theoutput order of a plurality of the view images according to Table 4below. The plurality of view images may include 2 view images.

Table 4 shows the output order of the view images, corresponding to themovement angle calculated as the gaze is moved to the right. In Table 4,the input image may be a 36-view image and a plurality of the displayedview images may include only 2 view images out of the 36 view images.

TABLE 4 Movement angle Output orders of view images (θ) 1 2 3 4 5 6 7 89 10 11 12 0 15 15 15 15 15 15 22 22 22 22 22 22 1 14 14 14 14 14 21 2121 21 21 21 14 2 13 13 13 13 20 20 20 20 20 20 13 13 3 12 12 12 19 19 1919 19 19 12 12 12 4 11 11 18 18 18 18 18 18 11 11 11 11 5 10 17 17 17 1717 17 10 10 10 10 10 6 16 16 16 16 16 16 9 9 9 9 9 9 7 15 15 15 15 15 88 8 8 8 8 15 8 14 14 14 14 7 7 7 7 7 7 14 14 9 13 13 13 6 6 6 6 6 6 1313 13 10 12 12 5 5 5 5 5 5 12 12 12 12 11 11 4 4 4 4 4 4 11 11 11 11 1112 3 3 3 3 3 3 10 10 10 10 10 10 13 2 2 2 2 2 9 9 9 9 9 9 2 14 1 1 1 1 88 8 8 8 8 1 1 15 1 1 1 7 7 7 7 7 7 1 1 1 16 1 1 6 6 6 6 6 6 1 1 1 1 17 15 5 5 5 5 5 1 1 1 1 1 18 4 4 4 4 4 4 1 1 1 1 1 1 19 3 3 3 3 3 1 1 1 1 11 3 20 2 2 2 2 1 1 1 1 1 1 2 2 21 1 1 1 1 1 1 1 1 1 1 1 1

According to Table 4, when the gaze of the user is moved to the rightand the calculated movement angle is about 12θ, both the left and theright eyes may be disposed in the sub-viewing zone. Accordingly, thedisplay 230 may display an output image comprised of a 3-view image anda 10-view image. As the output order determining unit 220 determines theoutput order referring to FIG. 4, the user located in the middle of thedisplay may first see 15 and 22-view images and further see up to a1-view image while moving the gaze to the right.

For example, when the gaze of the user is moved to the right by about 2θas shown in FIG. 18, the output order determining unit 220 maydetermine, based on Table 4, to output the view images in order of 13,13, 13, 13, 20, 20, 20, 20, 20, 20, 13, and 13-view images 1810.Therefore, even though the gaze is moved to the right by about 2θ,respectively two 13-view images may be continued both to the left andthe right of a 13-view image 1824 which corresponds to a position towhich the gaze is moved. In other words, the right eye may see the viewimages in order of 13, 13, 13, 13, and 13-view images instead of 13, 13,13, 11, and 12-view images.

Table 5 shows the output order of the view images, corresponding to themovement angle calculated as the gaze is moved to the left. In Table 5,the input image may be a 36-view image and a plurality of the displayedview images may include only 2 view images out of the 36 view images.

TABLE 5 Movement angle Output orders of view images (θ) 1 2 3 4 5 6 7 89 10 11 12 0 15 15 15 15 15 15 22 22 22 22 22 22 1 23 16 16 16 16 16 1623 23 23 23 23 2 24 24 17 17 17 17 17 17 24 24 24 24 3 25 25 25 18 18 1818 18 18 25 25 25 4 26 26 26 26 19 19 19 19 19 19 26 26 5 27 27 27 27 2720 20 20 20 20 20 27 6 28 28 28 28 28 28 21 21 21 21 21 21 7 22 29 29 2929 29 29 22 22 22 22 22 8 23 23 30 30 30 30 30 30 23 23 23 23 9 24 24 2431 31 31 31 31 31 24 24 24 10 25 25 25 25 32 32 32 32 32 32 25 25 11 2626 26 26 26 33 33 33 33 33 33 26 12 27 27 27 27 27 27 34 34 34 34 34 3413 35 28 28 28 28 28 28 35 35 35 35 35 14 36 36 29 29 29 29 29 29 36 3636 36 15 36 36 36 30 30 30 30 30 30 36 36 36 16 36 36 36 36 31 31 31 3131 31 36 36 17 36 36 36 36 36 32 32 32 32 32 32 36 18 36 36 36 36 36 3633 33 33 33 33 33 19 34 36 36 36 36 36 36 34 34 34 34 34 20 35 35 36 3636 36 36 36 35 35 35 35 21 36 36 36 36 36 36 36 36 36 36 36 36

Referring to Table 5, when the gaze is moved to the left and thecalculated movement angle is about 12θ, both the left and the right eyesmay be disposed in the sub-viewing zone. Accordingly, the display 230may display an output image comprised of a 27-view image and a 34-viewimage. As the output order determining unit 220 determines the outputorder referring to FIG. 5, the user located in the middle of the displaymay first see 15 and 22-view images and further see up to a 36-viewimage while moving the gaze to the left. In other words, the imageprocessing apparatus 200 enlarges the viewing angle twice to the left ofthe user, thereby supplying a natural 3D image without causing theinversion.

For example, when the gaze of the user is moved to the left by about 2θas shown in FIG. 19, the output order determining unit 220 may determineaccording to Table 5 to output the view images in order of 24, 24, 17,17, 17, 17, 17, 17, 24, 24, 24, and 24-view images 1910. Therefore, eventhough the gaze is moved to the right by about 2θ, respectively two24-view images may be continued both to the left and the right of a24-view image 1922 which corresponds to a position to which the gaze ismoved. In other words, the left eye may see the view images in order of24, 24, 24 (1922), 24, and 24-view images.

In the same manner, when the input image is a 12-view image and when thegaze is moved to the right, the output order determining unit 220 maydetermine the output order corresponding to the movement angle accordingto Table 6 below. Here, when the gaze is moved to the left, the outputorder determining unit 220 may determine the output order of the viewimage corresponding to the movement angle as shown in Table 6 below.

Table 6 shows the output order of the view images, corresponding to themovement angle calculated as the gaze is moved to the right. In FIG. 6,the input image may be a 12-view image and the plurality of displayedview images may include only 2 view images out of the 12 view images.

TABLE 6 Movement angle Output orders of view images (θ) 1 2 3 4 5 6 7 89 10 11 12 0 3 3 3 3 3 3 10 10 10 10 10 10 1 2 2 2 2 2 9 9 9 9 9 9 2 2 11 1 1 8 8 8 8 8 8 1 1 3 1 1 1 7 7 7 7 7 7 1 1 1 4 1 1 6 6 6 6 6 6 1 1 11 5 1 5 5 5 5 5 5 1 1 1 1 1 6 4 4 4 4 4 4 1 1 1 1 1 1 7 3 3 3 3 3 1 1 11 1 1 3 8 2 2 2 2 1 1 1 1 1 1 2 2 9 1 1 1 1 1 1 1 1 1 1 1 1

Referring to Table 6, when the input image is the 12-view image and themovement angle becomes about 9θ, the left eye and the right eye both seea 1-view image, that is, a 2D image is seen by the user. However,although the display 230 displays an output image comprised of twelve1-view images, the output image is seen as a 3D image by the user due tomotion parallax.

For example, when the gaze is moved to the right by about 2θ as shown inFIG. 20, the output order determining unit 220 may determine to outputthe view images in order of 01, 01, 01, 01, 08, 08, 08, 08, 08, 08, 01,and 01-view images 2010. Therefore, even though the gaze is moved to theright by about 2θ, respectively two 01-view images may be continued bothto the left and the right of a 01-view image 2024 which corresponds to aposition to which the gaze is moved. In other words, the right eye maysee the view images in order of 01, 01, 01 corresponding to 01 2024, 01,and 01-view images. In other words, in a display that displays 12 viewimages, the 01 and 01-view images may be rendered in a 11^(th) and12^(th) position 2025.

Table 7 shows the output order corresponding to the movement anglecalculated as the gaze is moved to the left. In FIG. 7, the input imageis a 12-view image and a plurality of the displayed view images mayinclude only 2 view images out of the 12 view images.

TABLE 7 Movement angle Output orders of view images (θ) 1 2 3 4 5 6 7 89 10 11 12 0 3 3 3 3 3 3 10 10 10 10 10 10 1 11 4 4 4 4 4 4 11 11 11 1111 2 12 12 5 5 5 5 5 5 12 12 12 12 3 12 12 12 6 6 6 6 6 6 12 12 12 4 1212 12 12 7 7 7 7 7 7 12 12 5 12 12 12 12 12 8 8 8 8 8 8 12 6 12 12 12 1212 12 9 9 9 9 9 9 7 10 12 12 12 12 12 12 10 10 10 10 10 8 11 11 12 12 1212 12 12 11 11 11 11 9 12 12 12 12 12 12 12 12 12 12 12 12

Referring to Table 7, since the input image is the 12-view image, whenthe movement angle is about 9θ, the left and the right eyes both seeonly the 12-view image, that is, a 2D image is seen by the user.However, although the display 230 displays an output image comprised oftwelve 12-view images, the output image is seen as a 3D image by theuser due to motion parallax.

For example, when the gaze is moved to the left by about 2θ as shown inFIG. 21, the output order determining unit 220 may determine to outputthe view images in order of 12, 12, 05, 05, 05, 05, 05, 05, 12, 12, 12,and 12-view images 2110 according to FIG. 21. Therefore, even though thegaze is moved to the left by about 2θ, respectively two 12-view imagesmay be continued to the left and the right of a 12-view image 2122 whichcorresponds to a position to which the gaze is moved. In other words,the left eye sees the view images in order of 12, 12, 12(2122), 12, and12-view images. That is, in a display that displays the 12 view images,the 12 and 12-view images may be rendered in a first and second position2125.

Tables 1 through 7 above may be stored in the image processing apparatus200 in the form of a database. Accordingly, the output order determiningunit 220 may determine the output order of the view images correspondingto the movement angle calculated by the movement angle calculating unit212, by referring to the table database.

FIG. 22 is a flowchart describing a process of determining an outputorder of view images according to the movement of a user's gaze.

Referring to FIG. 22, the movement estimation unit 210 may estimatewhether the gaze of the user is moved in operation S2210.

For example, the sensing device 100 may photograph the user in real timeand transmit the photographed image to the movement distance calculatingunit 211 in real time. Accordingly, the movement distance calculatingunit 211 may calculate positions of the left eye and the right eye ofthe user from the photographed image, and calculate next positions ofthe left or the right eye or of both eyes from a next photographedimage. Here, the left and right eye positions may be expressed ascoordinate values. The movement distance calculating unit 211 maycalculate difference values between current and next left eye positionsand between current and next right eye positions calculated from thecurrent and the next photographed images, thereby calculating the leftdifference value and the right difference value.

The movement distance calculating unit 211 may estimate whether the gazeis moved using the left and the right difference values, in operationS2220.

For example, when any of the left difference value and the rightdifference value is equal to or greater than the preset error value, themovement distance calculating unit 211 may estimate that the gaze ismoved (“YES” branch of operation S2220). In addition, when the left andthe right difference values are positive, the movement distancecalculating unit 211 may estimate that the gaze is moved to the right.When the left and right difference values are negative, the movementdistance calculating unit 211 may estimate that the gaze is moved to theleft. The movement distance x may be expressed as a coordinate valuecomprised of the left and the right difference values.

The movement angle calculating unit 212 may calculate the movement angleof the eye of the user, corresponding to the gaze movement of the user,using the movement distance x and a focal distance fl. For example, themovement angle calculating unit 212 may calculate the movement angleusing Equation 1 described above.

Next, when the gaze of the user is moved, the output order determiningunit 220 may determine the output order of the view images correspondingto a movement direction of the gaze, in operation S2230.

For example, the output order determining unit 220 may determine theoutput order of the view images corresponding to the movement anglecalculated with reference to Tables 1 through 7. More specifically, theoutput order may be determined based on the input image and the movementdirection of the gaze. That is, the output order of the view imagescorresponding to the movement angles may be determined according towhether the input image is a 12-view image or a 36-view image andwhether the gaze of the user is moved to the right or the left.

That is, the output order determining unit 220 may determine the outputorder of the view images corresponding to the left eye of the user whenthe gaze of the user is moved to the right. When the gaze is moved tothe left, the output order determining unit 220 may adjust the outputorder of the view images corresponding to the left eye. Here, the outputorder may be adjusted such that the left slide window and the rightslide window include respectively different view images consecutivelyarranged.

Sizes of the left and the right slide windows may be presetcorresponding to the number of view images that can be simultaneouslydisplayed in one pixel. Left and right view images of a view imagecorresponding to a position to which the gaze is moved may berespectively disposed in the middle of the left and the right slidewindows. In addition, a reference number of view images may be continuedto the left and the right of the view images disposed in the middle.Here, the reference number may be preset as a value obtained by dividingthe number of view images simultaneously displayed in one pixel by 2.Thus, by the left and the right slide windows, the natural 3D image maybe seen by the user even if the user's eyes are minutely moved to theleft and the right.

Next, the display 230 may display a plurality of view images accordingto the determined output order in operation S2240. That is, the display230 may display the output image comprised of a plurality of differentview images. Here, the output image may include pixel units eachincluding at least one view image. Therefore, even when the gaze of theuser is moved to the left or the right, the left eye of the user seesonly the left view images and the right eye sees only the right viewimages.

As another example, the output order determining unit 220 may adjust theoutput order using only 2 view images out of the plurality of viewimages. In this case, the display 230 may display the view imagecomprised of the two view images according to the adjusted output order.

When the gaze is estimated as not having been moved in operation S2210(“NO” branch of operation S2220), the display 230 may continuouslydisplay the output image according to the order of the view images beingdisplayed in operation S2250.

The display 230 described above may include a lens structure such as alenticular lens or a barrier structure.

The methods according to the above-described example embodiments may berecorded in non-transitory computer-readable media or processor-readablemedia including program instructions to implement various operationsembodied by a computer. The media may also include, alone or incombination with the program instructions, data files, data structures,and the like. The program instructions recorded on the media may bethose specially designed and constructed for the purposes of the exampleembodiments, or they may be of the kind well-known and available tothose having skill in the computer software arts.

Examples of non-transitory computer-readable media include magneticmedia such as hard disks, floppy disks, and magnetic tape; optical mediasuch as CD ROM disks and DVDs; magneto-optical media such as opticaldisks; and hardware devices that are specially configured to store andperform program instructions, such as read-only memory (ROM), randomaccess memory (RAM), flash memory, and the like. Examples of programinstructions include both machine code, such as produced by a compiler,and files containing higher level code that may be executed by thecomputer using an interpreter. The described hardware devices may beconfigured to act as one or more software modules in order to performthe operations of the above-described example embodiments, or viceversa. The methods may be executed on a general purpose computer orprocessor or may be executed on a particular machine such as the imageprocessing apparatus described herein. Any one or more of the softwaremodules described herein may be executed by a dedicated processor uniqueto that unit or by a processor common to one or more of the modules.

Although example embodiments have been shown and described, it would beappreciated by those skilled in the art that changes may be made inthese example embodiments without departing from the principles andspirit of the disclosure, the scope of which is defined in the claimsand their equivalents.

What is claimed is:
 1. An image processing apparatus comprising: aprocessor to control one or more processor-executable units; a movementestimation unit to estimate a movement of a gaze of a user; and anoutput order determining unit to determine an output order of aplurality of view images corresponding to the movement of the gaze ofthe user.
 2. The image processing apparatus of claim 1, wherein: themovement estimation unit estimates a movement direction of the gaze ofthe user; and the output order determining unit adjusts the output orderof the plurality of view images corresponding to a right eye of the userwhen the gaze of the user is moved to the left.
 3. The image processingapparatus of claim 1, wherein: the movement estimation unit estimates amovement direction of the gaze of the user; and the output orderdetermining unit adjusts the output order of the plurality of viewimages corresponding to a left eye of the user when the gaze of the useris moved to the right.
 4. The image processing apparatus of claim 1,wherein: the movement estimation unit estimates a movement angle of thegaze of the user; and the output order determining unit varies thenumber of the plurality of view images of which the output order is tobe adjusted, based on the movement angle.
 5. The image processingapparatus of claim 4, wherein, as the movement angle increases, theoutput order determining unit increases the number of the plurality ofview images of which the output order is to be adjusted.
 6. The imageprocessing apparatus of claim 1, further comprising a display to displayan output image comprising a plurality of the view images, according tothe determined output order.
 7. The image processing apparatus of claim6, wherein the output image comprises pixel units each containing atleast one view image.
 8. The image processing apparatus of claim 1,wherein the output order determining unit determines the output order ofthe plurality of view images such that a predetermined number of leftand right view images are consecutively arranged with respect to areference view image, the reference view image corresponding to aposition to which the gaze of the user is moved.
 9. An image processingmethod comprising: estimating a movement of a gaze of a user; anddetermining, by way of a processor, an output order of a plurality ofview images corresponding to the movement of the gaze of the user. 10.The image processing method of claim 9, wherein: the estimating of thegaze movement estimates a movement direction of the gaze of the user;and the determining of the output order adjusts an output order of theplurality of view images corresponding to a right eye of the user whenthe gaze of the user is moved to the right.
 11. The image processingmethod of claim 9, wherein: the estimating of the gaze movementestimates a movement direction of the gaze of the user; and thedetermining of the output order adjusts an output order of the pluralityof view images corresponding to a left eye of the user when the gaze ofthe user is moved to the left.
 12. The image processing method of claim9, wherein: the estimating of the gaze movement estimates a movementangle of the gaze of the user; and the determining of the output ordervaries the number of the plurality of view images of which the outputorder is to be adjusted, according to the movement angle.
 13. The imageprocessing method of claim 12, wherein the determining of the outputorder increases the number of the plurality of view images of which theoutput order is to be adjusted as the movement angle increases.
 14. Theimage processing method of claim 10, further comprising displaying anoutput image comprising a plurality of the view images according to thedetermined output order.
 15. The image processing method of claim 14,wherein the output image comprises pixel units each comprising at leastone view image.
 16. The image processing method of claim 9, wherein thedetermining of the output order determines the output order of theplurality of view images such that a predetermined number of left andright view images are consecutively arranged with respect to a referenceview image, the reference view image corresponding to a position towhich the gaze of the user is moved.
 17. A non-transitory computerreadable recording medium storing a program to cause a computer toimplement the method of claim
 9. 18. A method of adjusting a multi-viewimage in a multi-view display, the method comprising: detecting movementof a gaze of a user; and reordering, by way of a processor, an outputorder of a plurality of view images making up the multi-view image inaccordance with the detected movement of the gaze.
 19. A non-transitorycomputer readable recording medium storing a program to cause a computerto implement the method of claim
 18. 20. A multi-view image displayapparatus displaying a multi-view image, the apparatus comprising: aprocessor to control one or more processor-executable units; a movementestimation unit to estimate a movement of a gaze of a user; and anoutput order determining unit to change an output order of a pluralityof view images making up the multi-view image in accordance with themovement of the gaze estimated by the movement estimation unit.